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Abstract: Accident Precursor Analysis (APA) serves as the bridge between existing risk modeling 
activities, which are often based on historical or generic failure statistics, and system anomalies, which 
provide crucial information about the failure mechanisms that are actually operative in the system. 
APA does more than simply track experience: it systematically evaluates experience, looking for 
under-appreciated risks that may warrant changes to design or operational practice. This paper 
presents the pilot application of the NASA APA process to Space Shuttle Orbiter systems. In this 
effort, the working sessions conducted at Johnson Space Center (JSC) piloted the APA process 
developed by Information Systems Laboratories (ISL) over the last two years under the auspices of 
NASA’s Office of Safety & Mission Assurance, with the assistance of the Safety & Mission 
Assurance (S&MA) Shuttle & Exploration Analysis Branch. This process is built around facilitated 
working sessions involving diverse system experts. One important aspect of this particular APA 
process is its focus on understanding the physical mechanism responsible for an operational anomaly, 
followed by evaluation of the risk significance of the observed anomaly as well as consideration of 
generalizations of the underlying mechanism to other contexts. Model completeness will probably 
always be an issue, but this process tries to leverage operating experience to the extent possible in 
order to address completeness issues before a catastrophe occurs. 
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1. INTRODUCTION 

The present NASA APA process [1] was developed through an iterative cycle of methodology 
development and pilot application, beginning with the Nuclear Regulatory Commission (NRC) 
process [2] as a point of departure, and augmenting that process as necessary based on fundamental 
differences in the nature and objectives of accident precursor analysis at NASA relative to the NRC. 
The resulting approach was tested and refined as necessary based on a number of concrete exercises. 
In all, three 3-day pilot accident precursor analysis (PAPA) working sessions were conducted at the 
Johnson Space Center (JSC). The working sessions reviewed Corrective Action Reports (CARs) and 
numerous anomaly reports from Space Shuttle missions flown from 2005 through 2007, utilizing the 
tools and expertise of the Safety and Mission Assurance (S&MA) Problem Investigation Team (SPIT) 
and the JSC S&MA broader community as well as various systems engineers. The specific approaches 
employed, as well as some of the key highlights from these working sessions and how they shaped the 
current NASA APA process, will be discussed in this paper. 

The first PAPA working session (the only one not conducted in the SPIT room) focused on a sampling 
of Shuttle Orbital Maneuvering System (OMS) and Reaction Control System (RCS) anomalies from 
the CAR database. The second PAPA working session was conducted utilizing the tools and expertise 
of the SPIT, focused on anomalies from all systems for Space Shuttle flights STS-1 14, STS-121, STS- 
115, and STS-1 16 (all flights conducted in 2005 & 2006). The third PAPA working session, also 
conducted in the SPIT room, focused on anomalies from all systems for Space Shuttle flights STS- 
1 17, STS-1 18, & STS- 120 (all flights conducted in 2007). 
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All three of these exercises have contributed to the refinement of what is the current APA process. The 
JSC exercises provided venues for refining the anomaly evaluation and grading process in terms of the 
computer-based forms used to structure the sessions and record the technical interactions, establishing 
the set of deterministic screening criteria that began to emerge from the hand-on experience, and 
structuring the mode of facilitation of the sessions themselves. 

The exercises used a multidisciplinary team environment similar to that used to conduct Hazard and 
Operability Studies (HAZOPs). The environment is that of structured brainstorming. At a minimum, 
the evaluation and grading teams were composed of accident precursor analysis experts to act as 
facilitator and scribe, subsystem experts who understand the anomalies and subsystems under analysis, 
and risk analysts who understand the subsystem- and system-level effects of failures. The teams 
documented evaluation and grading exercises in a purpose-built form which also captures what, if any, 
further analysis the team believes is warranted based on their grading and the caliber of the evidence 
used in making that determination. The specifics of how grading is conducted and how evidence is 
integrated in the final results are discussed in Ref [1]. 

2. SUMMARY OF APA WORKING SESSION PROCEEDINGS 

2.1. First JSC PAPA Working Session: November 13-16, 2007 

The objectives of this working session were: 

1. To demonstrate a pilot application of an early version of the APA technical approach; 
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2. To introduce the Space Shuttle S&MA organization to a practical application of APA 

3. To effectively demonstrate the utility of the precursor process in meeting S&MA’s objectives 

4. To learn from the pilot application with the purpose of continuing to refine and improve the 
APA approach 

This first working session focused on CARs associated with OMS/RCS. The CAR database records all 
anomalous conditions, regardless of hardware life-cycle, or severity of condition. This means that the 
record of CARs is more likely to include anomalies of negligible risk implications than some other 
data sources [3]. CARs were chosen for this exercise because the Space Shuttle S&MA Office had 
already collected a summary of CARs that they had reviewed for Probabilistic Risk Assessment (PRA) 
updating purposes. Using only CARs is not ideal from an APA standpoint, because there may have 
been anomalies reported in other databases that could ultimately be identified as precursors but for 
which no CAR was written. However, the CARs were adequate for the purposes of our first pilot 
exercise, since the focus was more on process rather than the adequacy of the data being processed. 

In CARs, the data record for a given anomalous event is developed over time. It is initiated at the time 
of its observation, at which point the descriptive information for the event is entered into the database. 
Subsequently, as the problem is managed, additional information relating to causal analysis, corrective 
action, and other pertinent information is added. 

This raised the issue of when, in the APA process, to interface with the data in order to provide the 
greatest value. For example, given the process’ emphasis on generalizing from the causal failure 
mechanism, it would be valuable to wait for a causal analysis to be performed on the reported 
anomalous event, assuming that this is a prescribed part of the existing corrective action process. 
However, there is a non-zero risk associated with any delay in identifying a potential accident 
precursor, since it might extend the window of opportunity for the failure mechanism to result in 
severe consequences. As a practical matter, data interface issues must be resolved on a case-by-case 
basis depending on the details of the reporting system being used for input. As will be discussed in the 
next section, the second exercise was carried out on Mission Evaluation Room (MER) anomalies [4], 



with CARs used to inform the evaluation and grading exercise when available. The cumulative 
experience from both exercises suggests strongly that understanding the “anomaly failure mechanism” 
is an extremely important ingredient for the NASA APA process. As part of the process of identifying 
under-appreciated risks, the "generalization" step of APA systematically considers the potential for 
each observed failure mechanism to operate in different locations and at different severity levels. 
Without an understanding of the failure mechanism, generalization, which is a key step, would be 
almost entirely speculative. This is not simply to say that the process should wait for information 
about the failure mechanism, but to add that failure mechanism information is important to capture, 
and it’s important that it be correctly identified. However it should be noted that analysis of some 
items in all of the JSC exercises was hampered by lack of information about the mechanism. 

A preliminary edition of what became known as the anomaly potential precursor deliberation (APPD) 
form used to capture the results of the exercise was introduced in this first exercise. The preliminary 
form added significant value relative to not having a form, but the working sessions also motivated 
significant changes which were incorporated before the next working session was held. 

The main aspects of the APA approach planned and conducted for this exercise are shown in Figure 1. 
The steps outlined in dotted lines were part of the overall APA approach but were not carried out in 
this particular working session. In the cases where a “(TBD)” is included in the box, the hope was that 
criteria would present themselves during the working session that could be instantly applied or 
developed for subsequent working sessions. This hope was realized in the first two working sessions 
held at the JSC in regards to “rule-based screening” criteria and some progress was made in regards to 
“automated screening” criteria in the third working session as will be discussed later in the paper. The 
“Observation and Trending” box is also labeled “TBD” because although anomalies or their 
generalizations may be graded for “Observation and Trending,” this process has yet to be fully defined 
and integrated. Trending analysis is already conducted within the Space Shuttle Program, but a full 
understanding of the approach employed did not fully present itself over the course of the three 
working sessions, and thus it was not integrated into the overall APA process. This is one of the 
aspects of APA that continue to be investigated for future development and integration. 


Figure 1. APA Technical Approach for 1st JSC PAPA Working Session 








For this first working session, each CAR was reviewed in succession with an attempt made to identify 
a failure mechanism based on the information in the CAR, followed by the definition of a “failure 
condition of concern”. Ideally, the “failure condition of concern” should be a specific hard component 
failure, essentially equivalent to a basic event in the system’s PRA model (if the PRA model addresses 
the subject area); in practice, the “failure condition of concern” could be any condition that was 
serious enough to concern the system experts. The key is to define it with an adjective or explicit 
threshold (i.e. a pressure loss greater than 2 psi/min) that (a) provides a means of comparing the actual 
anomaly that occurred against the “failure condition of concern,” so that a Failure Condition Index 
(FCI) can be assigned to qualitatively assess the likelihood of the failure condition in the presence of 
the failure mechanism; and (b) provides a clear starting point for assignment of a Conditional 
Consequence Index (CCI) which qualitatively assesses the likelihood of a severe consequence given 
the occurrence of the failure condition of concern. 

In some cases, a failure condition of concern could be defined without a specific failure mechanism 
identified, however in such cases, generalizations of the anomaly could not be postulated. These cases 
should be revisited to ensure that a potential precursor is not being overlooked simply because there is 
a dearth of information. 

For each anomaly, an FCI and CCI were assigned to determine the overall Potential Problem Index 
(PPI) for the anomaly. The assignment of an FCI and CCI, although simple in principle, involves 
detailed review of evidence and concerted deliberations of what the evidence is telling you. These 
deliberations are guided by the process facilitator, who plays the role of discussion orchestrator to 
ensure that the proper details are solicited from the system experts and recorded in the APPD as well 
as ensuring that the evidence being used to assign the FCI and CCI is properly recorded. Therefore the 
entire team would discuss the selection of proper FCI and CCI as well as the evidence that these were 
based on. In this first working session, only one set of evidence was collected for both FCI and CCI. 
This was changed in the second and third working sessions, with each factor receiving its own 
evidence assessment so that the index and the supporting evidence were explicitly linked. 

In the course of this first JSC working session, it emerged that many active failures in this system can 
be screened out, based on their character and on the fault tolerance built into the OMS/RCS functions. 
During the session, the following rule-based screen was promulgated: 

Failures can be screened out if: 

They are within design fault tolerance (of OMS/RCS, required to be dual fault tolerant) 
AND have NO: 

• Common cause failure (CCF) potential 

• Downstream or upstream functional effects (cascading downstream or upstream 
failures having broader impacts) (e.g., failure of a pressure regulator could lead 
to overpressure leading to leak) 

• No phenomenological failures that affect the surrounding hardware (e.g., a 
hydrazine leak could affect nearby components even if the loss of hydrazine per 
se was not critical to the system from which it was leaking). 

Note that this rule is somewhat analogous to one of NRC’s screening rules: short-term loss of 
redundancy in only one system [2]. The rule noted above was then a starting point for the second 
working session addressing a broader class of anomalies (i.e., anomalies beyond OMS/RCS), although 
in the second working session, it was found necessary to specify actual redundancy, rather than simply 
being “within fault tolerance.” 



On the last day of the first working session, the team decided to attempt an “Analysis” exercise for one 
of the anomalies graded for “Risk Modeling”. During the course of this exercise, it was found that this 
failure scenario was not modeled in the Space Shuttle PRA [5], but had been considered and screened 
out of the model on probabilistic arguments. Therefore, the finding of this exercise was simply 
“ further analysis leads to reconsidering inclusion of this scenario in the PRA model ” Since the PRA 
model is updated in block changes, this could be done as part of one of the future block changes. 

In order to address the matter of interfacing between “Evaluation & Grading” and “Risk Modeling”, it 
was decided that the Evaluation & Grading Team should document the result of a recommendation for 
analysis in the form of a Summary of Concern (SOC) which the PRA team could then use to initiate 
their own PRA reviews - in effect carrying out the “Analysis” part of the APA process twice; once in 
a perfunctory qualitative manner in the SOC and then in a more rigorous quantitative manner using the 
PRA. 

As part of the first working session closeout, the ISL team conducted a “Pause and Learn” feedback 
session to solicit inputs and comments from the working group. These inputs were used to further 
refine the technical approach as well as modify the APPD for the second working session, as discussed 
in the next section. 

2.2 Second JSC PAPA Working Session: March 3-7, 2008 

The second working session was different from the first in some key aspects: 

• The exercise was based on MER anomalies for all Space Shuttle Orbiter systems from four 
consecutive missions (from Return-to-F light onward), rather than being focused on one 
system’s incidents (OMS/RCS) over the life of the program. 

• The APPD form was significantly enhanced as mentioned in the previous section. 

• The exercise was carried out in the SPIT room at JSC, where engineering and general safety 
information is readily available. 

• Several “SPIT Bosses” were involved in the exercise. These are highly knowledgeable 
individuals who not only know a great deal about the Shuttle, but also had been personally 
involved in dealing with most or all of the subject anomalies as they arose in-flight. 

The main aspects of the technical approach utilized for this exercise are shown in Figure 2. Significant 
new elements are: 

• a nascent “Rule-Based Screen” was put in place and was further refined during this exercise, 
as will be discussed; 

• evidence is now collected for both the assignment of FCI and CCI, which are combined to 
provide an overall evidence caliber (EC) for the PPI; 

• a “Summary of Concerns” (SOC) was written for each anomalous event if it, or any of its 
generalizations, was graded for “Risk Modeling” rather than making comparisons directly 
against the Space Shuttle PRA scenarios. Note that the SOC was later abandoned in the 
current APA process and replaced by the development of an Enhanced Functional Flow Block 
Diagram (EFFBD) approach as discussed in Ref [1]. 



Figure 2. APA Technical Approach for 2nd JSC PAPA Working Session 



During the second working session, rules also began to emerge for application of the PPI and EC. 

1. Requirement for evidence to justify low likelihood: To justify assigning “unlikely” for FCI or 
CCI, you should have at least 2 types of evidence. 

2. Do not cite “operational experience” as evidence that something will NOT happen. 
“Operational experience” was introduced, at an earlier stage of process development, as an 
evidence type capable of proving that something CAN happen. Using experience to argue that 
something CANNOT happen would tend to normalize deviance. 

There were 125 documented MER anomalies that occurred during the 4 flights chosen for this 
exercise. For the purposes of time management, a subset of these (roughly half) were chosen for this 
working session. These were organized according to Space Shuttle system so that system experts 
could be scheduled to participate based on need. The half that were chosen were not “pre-screened” 
for relevance per se; all the MER anomalies are deemed relevant, since they were flight issues 
considered serious enough to review in the SPIT room during the flight. In addition to the scheduled 
participation of various system experts, there were SPIT bosses (2 to 3) that participated throughout 
the working session. SPIT bosses are the lead personnel that evaluate anomalies during a flight and 
serve a triage-like function by deciding which anomalies need further review by system specialists and 
which can be closed out as non-issues'. They are also Space Shuttle generalists and served as both 
points of information on specific systems as well as providing insights into how a failure mechanism 
may be generalized across systems. Space Shuttle S&MA staff and ISL APA staff rounded out the 
group. 

A revised version of the Anomaly Potential Precursor Deliberation (Ver. 2 of the APPD) form was 
used to guide the discussion and record the various parts of the Evaluation & Grading process. After 


f A non-issue during a flight may still be deemed a potential precursor but was not flagged as a 
concern during the flight because of the low intensity or advantageous timing of the incident. 



reviewing the anomaly information as it was documented during the flight, the team would pull up the 
associated CAR, if one existed, to get a better understanding of the cause of the anomaly so that a 
failure mechanism could be identified, as well as recording the operational phase (Pre-Flight, Flight, 
Post-Flight) during which the failure mechanism became active or was realized. In some cases, 
particularly if no CAR had been written for the anomaly, it was deemed that “insufficient information” 
was present to disposition the anomaly and no failure mechanism could be identified. Note that this 
issue did not arise during the first working session, because we were using CARs as the anomaly data 
source. 

In some cases, if the anomaly was serious enough, a “failure condition of concern” could be defined 
without “sufficient information;” but without a specific failure mechanism, generalizations of the 
anomaly could not be postulated. In other cases without a failure mechanism, a “failure condition of 
concern” could not even be defined, since it was unclear how the anomaly could be exacerbated, and 
in cases where a “failure condition of concern” could be defined, an FCI could not be defined because 
it was unclear how likely the mechanism is to lead to the “failure condition of concern”. In actuality, 
there were not too many cases where there was “insufficient information” but, at the time it was 
assumed that these cases would be revisited to ensure that a potential precursor is not being 
overlooked simply because there is a dearth of infonnation. 

The rule-based screen identified in the first working session was included in Version 2 of the APPD to 
rapidly screen out anomalies that met that criterion. This screen proved effective and more criteria 
were added during the course of this working session. For instance, some of the MER anomalies stem 
from instrumentation having essentially no safety relevance. Accordingly, a rule was developed 
permitting screening an anomaly out if: 

• it pertained to monitoring instrumentation only; 

• the instrumentation was not used for real-time decision-making; 

• the instrumentation was not used for closed-loop control; 

• the instrumentation was not for caution, warning, or redundancy management. 

If an anomaly was not screened out by the preset criteria, then a FCI and CCI were assigned to 
determine the overall PPI for the anomaly as described in the previous section. 

The issue of whether something was modeled in the PRA came up intermittently. If the S&MA 
representative claimed that it was modeled, then the question became “Should we represent the CCI to 
be a function of the PRA results, or should we claim that it is sufficiently modeled in the PRA and 
skip the assessment of the anomaly, since we are ultimately interested in anomalous conditions that 
indicate (after evaluation and grading) that the likelihood of the failure or its propensity to lead to 
severe consequence are underappreciated. The team seemed to settle on the notion that PICA results 
would be used when possible to represent the CCI assignment with the “reliability analysis” that was 
part of the PRA being called the evidence for the assignment. 

In some cases, the hazard analysis for the “failure condition of concern” was examined and its results 
used as evidence in assigning the CCI. However, with experience, it became clear that not all hazard 
analyses were carried out with the same level of rigor and, of more concern, in a few cases, attempts 
were made to invoke the absence of a hazard analysis as evidence that the CCI assignment should be 
low. Unfortunately, the lack of a hazard analysis could simply be an indication that a “failure 
condition of concern” is not fully appreciated and thus could indeed lead to serious consequences; 
therefore, changes were made in the role and importance that hazard analyses play in the evaluation & 
grading step within the current APA process. 

By the end of the working session, the regular attendees (basically everyone but the system experts) 
had become familiar with the main aspects of the process. For instance, the SPIT bosses would start 
looking up evidence as the deliberations were taking place (since they knew that the facilitator would 
ultimately ask them what evidence they used in coming to their conclusions), and the S&MA staff 



would start reading through hazards or looking up parts of the PRA model in anticipation that this 
information would prove useful in the assignment of CCI. This, of course, made the conduction of the 
process much more efficient and effective. In order to produce consistent results, it was important that 
the facilitator insist that ONLY the data that could be provided real-time and verified be recorded as 
evidence in the APPD. The point of this was that, if someone claimed that some evidence existed but 
they could not produce it then this would be reflected in the estimation of the EC used in assigning 
FCI and CCI. If a low EC caused the anomaly to be graded for further analysis, then the data that were 
thought to exist could be found and reviewed and the additional evidence could be included in the 
APPD to see if it affected the way that the anomaly had been graded. In the current form of the APA 
process the evidence gathering is called out as a stand-alone step such that the evaluation & grading 
team is not hampered by having to search for the data real-time. 

The percentage breakdown of the Evaluation & Grading results for the anomalies and generalizations 
that were considered during the course of the working session are shown in Figure 4. Note that the 
“Rule-Based Importance Screen” refers to the rule-based screening criteria available at the time. 

2.3 Third JSC PAPA Working Session: April 22-24, 2009 

The third working session was similar to the second working session but differed in a few key areas: 


• The exercise was based on both MER anomalies and Problem Reporting and Corrective 
Actions (PRACA) In-flight Anomalies (IFAs) [6] for the three missions flown in 2007. 


• The APPD form used had undergone another major revision to improve the user interface for 
both entering information and displaying data based on the input from the second working 
session participants. 

The main aspects of the technical approach utilized for this exercise are shown in Figure 3. 


Figure 3. APA Technical Approach for 3rd JSC PAPA W orking Session 









Shortly into the working session, a number of MER records were encountered which were never 
elevated to anomaly status (they had an S&MA number but no MER number). In these cases it was 
found that the record contained an incident which had at first been reported as a potential anomaly but 
upon further review it was found that the system was behaving nominally. These records were 
screened out and were the basis for an automated screening rule for the rest of the session: 

IF a MER record had an S&MA number but no MER number then it is screened out 

Note that in Figure 3 we indicate that the automated screen is used but the box still contains a “(TBD)” 
because the screening criteria was identified during the working session but could instantly be applied. 
We, of course, also remained on the alert for any other automated screening criteria that could be used. 

The rule-based screen, first implemented in version 2 of the APPD (PAPA Working Session 2) and 
refined for version 3 (used in this working session), was used to rapidly screen out anomalies that met 
those criteria. This screening criteria proved rather effective and is expected to be implemented in 
future applications of the APA process. 

In many cases, the FMEA for the component analyzed was pulled up to reference the criticality of the 
anomalous hardware, as well as the fault tolerance in place to avoid severe consequence in the case of 
a component failure. The FMEA was particularly useful in assessing the CCI. Prior to the working 
session the APPD form was configured such that the first field for capturing evidence used to 
determine CCI would default to FMEA, in order to guide the evaluation into a semi-standardized 
usage of FMEA data for this step. In most cases, it was successful and the FMEA was used for the 
CCI assessment. 

An interesting issue that arose during the working session came about when the team started reviewing 
the Thermal Protection System (TPS) anomalies. Many of these some anomalies were graded for 
“Observation and Trending” due to the real-time inspection and on-orbit repair practices enacted 
following the Columbia accident [7] whereas these types of anomalies would have previously been 
graded for “Risk Modeling” without these controls as had occurred during the second working session 
before confidence had been gained in these control methods. This difference between grading results 
has spawned some discussion regarding the credit that should be taken for “corrective actions” 
implemented after an anomaly has been found to be risk significant but before the corrective action 
has fully demonstrated its effectiveness - this will be fodder for future considerations of how such 
evidence should be used in the APA process. For the time being however the process worked as 
expected - with a “caution flag” going up so that tracking continues in lieu of further risk modeling. 

3. SUMMARY OF APA WORKING SESSION RESULTS 

The screening and grading results from each of 
the three PAPA working sessions are shown in 
Figure 4. 

Note that although the time spent conducting 
each working session was about the same, the 
number of anomalies processed increased by 
about 100% from the first exercise to the third. 

This was due in part to the fact that some of 
the same people participated in all of the 
working sessions so they naturally became 
more adept at implementing the APA 
technical approaches. However another 
driving factor was the automated and rule- 
based screening criteria developed over the 
course of the three working sessions; in the 
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Figure 4. PAPA Working Session Results 
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last working session these accounted for 70% of the anomalies processed. 

Another interesting trend is the percentage of anomalies or related generalizations that were graded for 
“Risk Modeling”. Although it looks as if the percentage is going down, if the “screened-out” 
anomalous conditions are not considered then the percentage of items graded for “Risk Modeling” is 
20% for working sessions 1 & 2 but drops to 5% in working session 3. As explained in the body of the 
paper, this is in part due to the perception that TPS anomalies, even if they were to progress to a 
“failure condition of concern”, were much less likely to cause a severe consequence (burn-through 
during re-entry in this case) because of the on-orbit inspection and repair methods that were 
incrementally put in place mainly between working sessions 2 & 3. As noted earlier these anomalies 
were instead graded for “Observation & Trending” which can be seen as a percentage increase in this 
grading category between working sessions 2 & 3. This highlights the fact that although the perception 
was that the corrective action had been effective; the process still advocates a “watch and see” 
outcome regarding the ultimate verdict of the actual effectiveness. 

4. CONCLUSIONS 

This paper discussed the conduction and results of three pilot accident precursor analysis (PAPA) 
working sessions held at NASA’s Johnson Space Center (JSC) between November 2007 and April 
2009. During these PAPAs over 150 anomalies were assessed in terms of their degree of risk 
significance to the safe operation of the Space Shuttle Orbiter using various early versions of what 
ultimately became the current NASA APA process [1]. 

The working sessions proved to be valuable exercises in helping to highlight areas of the APA 
methodology which required additional thought and modification. The exercises also informed the 
practical aspect of applying the process in real-world circumstances which helped shape the tools and 
methods utilized in conducting the generalization and grading deliberations that are part of the APA 
process today. 
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